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Abstract 

A general two Higgs doublet model (2HDM) is adopted to study the signature of flavor changing 
neutral Higgs (FCNH) decay —>• tc + tc , where <jP could be a CP-even scalar ( H° ) or a CP-odd 
pseudoscalar (A 0 ). Measurement of the light 125 GeV neutral Higgs boson (h°) couplings at the 
Large Hadron Collider (LHC) favor the decoupling limit or the alignment limit of a 2HDM, in 
which gauge boson and diagonal fermion couplings of h° approach Standard Model values. In such 
limit, FCNH couplings of h° are naturally suppressed by a small mixing parameter cos (/3 — a), 
while the off-diagonal couplings of heavier neutral scalars f 0 are sustained by sin(/3 — a) ~ 1. 
We study physics background from dominant processes with realistic acceptance cuts and tagging 
efficiencies. Promising results are found for the LHC running at 13 or 14 TeV collision energies. 
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I. INTRODUCTION 


The Standard Model (SM) is very successful in explaining almost all experimental data to 
date, culminating in the recent discovery of the long awaited Higgs boson at the CERN Large 
Hadron Collider (LHC) 0,0- In the SM, all elementary particles acquire mass from a single 
Higgs doublet that generates spontaneous electroweak symmetry breaking (EWSB). All 
charged fermions have their masses and Yukawa couplings to the Higgs boson as correlated 
but free parameters. Furthermore, there are no flavor changing neutral currents (FCNC) 
mediated by gauge interactions, nor by Higgs interactions (FCNH), at the tree level. The 
most important goals of the LHC, at Run 2 and beyond, are the study of Higgs properties 
and the search for signals, direct or indirect, of new physics beyond the SM. 

As the most massive particle ever discovered, the top quark might provide clues to better 
understand the mechanism of EWSB. A possible explanation for its heaviness could be 
provided by a special two Higgs doublet model for the top quark (T2HDM) 0, where 
it is the only fermion that couples to a Higgs doublet with a large vacuum expectation 
value (VEV). The second heaviest particle is the newly discovered Higgs boson (h°). With 
rriho < rn t , it opens up the possibility of top quark decays into h° plus a charm quark. The 
branching fraction of t — y ch° in SM at one loop level is approximately 3 x 10 “ 1 5 0-0 for 
rriho — 120 GeV. If this decay is detected, it would indicate a large effective FCNH coupling 
of tree-level origins 0, or very large enhancement from beyond SM loop effects 0. 

In flavor conserving two Higgs doublet models, a discrete symmetry 0-10] is often im¬ 
posed to distinguish the SU(2) doublet fields </>i from <f> 2 . Without such a discrete symmetry, 
a general two Higgs doublet model (2HDM) should possess FCNH vertices. To study such 
interactions, we adopt the following Lagrangian involving Higgs bosons and fermions [l ll. Il2l] . 

£y = -^ ^ F { [hi F S/ 3 - a + p F c^- a ] h° + [n F cp- a - pFsp-a] H° - i sgn(Q F )//A 0 } P r F 


F=U,D,L 


-u [ Vp D P R - p u Wp l ] DH + - V [p L P R ] LH + + H.c., 


( 1 ) 


where Pl,r = (1 T 75 )/ 2 , cp- a = cos(/? — a), sp- a = sin(/3 — a), tan/3 = v 2 /v\, and a 
is the mixing angle between neutral Higgs scalars in the Type II (2HDM-II) notation 0. 
k matrices are diagonal and fixed by fermion masses to n F = \f2mplv with v — 246 GeV, 
while p matrices are free and have both diagonal and off-diagonal elements. We adopt a CP 
conserving Higgs model and choose p matrices to be real but not necessarily Hermitian. U, 
D, L and v are vectors in flavor space (U = (u,c,t), etc.). h° and H° are CP-even scalars 
{rrih < run), while A 0 is a CP-odd pseudoscalar. 

With the advent of the LHC, theoretical interest in search of FCNH top decays (t —y ch °) 
picked up jl3-16], and the ATLAS and CMS experiments have already placed the branching 
fraction limit B(t —> ch °) < 5.6 x 10~ 3 0 , implying 


'htc + X hct <0.14. For LHC at Js = 
14 TeV and integrated luminosity of L = 3000 fb 1 , the ATLAS experiment expects 18j to 


reach B(t —$■ ch °) < 1.5 x 10 


-4 


i.e. 


probing down to A htc = Pt c cos(/3 — a)/\/2 < 0.024. 


The flavor changing heavy Higgs decay (H° —$■ tc + tc ) is complementary to FCNH top 
decay (t —>■ ch°), since the coupling Xh tc is proportional to cos (0 —ad while A Htc oc sin fd — a). 
Higgs boson data from the LHC favor the decoupling limit [19| or the alignment limit |20|,|21| 
of a 2HDM. In this limit, FCNH couplings of h° are naturally suppressed by small cos(/3 — a), 
while off-diagonal couplings of H°, A 0 are sustained by sin(/3 — a) ~ 1. 

In this letter, we study the discovery potential of the LHC in the search for heavy Higgs 
bosons H° or A 0 that decay into a top quark and a charm quark. The top quark then 
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decays into a b quark, a charged lepton (e or /i), and a neutrino. Taking LHC Higgs 
data and B physics constraints into account, we evaluate production rates with full tree- 
level matrix elements for both signal and background. We optimize the acceptance cuts 
to effectively reduce the latter with realistic 6 -tagging and mistag efficiencies. Promising 
results are presented for the LHC with i/s = 13 TeV as well as 14 TeV. 


II. CONSTRAINTS FROM DATA 

In this section, we apply the latest results from LHC Higgs measurements, as well as from 
B physics, to constrain the parameters p tt , Pbb, Pet, Ptc, and cos (/3 — a) of a general 2HDM 
that are relevant for observing flavor changing decays of heavy Higgs bosons at the LHC. 


A. Constraints from ATLAS and CMS 


Run 1 of LHC at = 7 and 8 TeV has provided us with information on the couplings of 
the Higgs boson h°, by measuring the event rates relative to the SM signal strength. Even 
with our general 2HDM, the light Higgs boson h° constitutes a narrow resonance, and the 
signal strength for a production channel X and final state Y can be written as 


<7 (X)B(hP^Y) 
&sm{X) B(h —> Y) S m 


( 2 ) 


ATLAS and CMS often show the signal strength of measurements in two dimensions 
by grouping gluon fusion and tth production on one axis (ggF), and vector boson fusion 
and associated production on the other axis (VBF). These contours can be used to draw 
constraints on 2HDM’s ( 22 l |. We follow a simpler approach and consider signal strengths for 
final states with the largest statistics, namely 77 , ZZ *, WW*, and rr, where the dominant 
production mode is gluon fusion, as well as the signal strength for the bb final state from the 
associated production Vh° with V = W or Z. Table [Q shows the average signal strengths 
obtained by the experimental groups in Run 1 . We combine the ATLAS and CMS results 
and show both the combined values with their uncertainties in the last column. 

To fold the allowed regions of 2HDM parameter space compatible with ATLAS and CMS 
data, we take the discovered Higgs boson as the lightest CP even state ( 6 °) of a general two 
Higgs doublet model and scan over the following sets of parameters: 


General 2HDM: cos(^ - a), p tt , p bb , p cc , p TT , 
Type II 2HDM: cos(/3 — a), tan (3 . 


In SM, the most important contributions to the Higgs total width come from bb, WW*, 
gg , rr, cc and ZZ*. The same channels are expected to contribute in a general 2HDM. In 
our analysis, we include all the relevant parameters in Eq. (|3]) that affect the total width, 
gluon fusion cross section, and the associated Higgs production (Vh°). 

We cover a wide range of values for each free parameter, and require that all Yukawa cou¬ 
plings of the mass eigenstates (h°, H°, A 0 , TY ± ) stay perturbative, and that the constraints 
given in Table |T] are satisfied. The signal strength for each production and decay channel 
can be expressed in terms of scale factors which are couplings of the Higgs boson to fermions 


3 



Final state 

p(ATLAS) 

p(CMS) 

p(comb.) 

h° — > 77 

i i 7+0.27 
1,1 ' —0.27 

H 


0 

1.16 

± 

0.18 

h° -> ZZ* -> U 

1 44+ 0 - 40 
i - 44 -0.33 

25] 

0 - 93 ±g;» 

26] 

1.13 

± 

0.22 

h° -»• WW* -> Mi' 

1 09 +0 ’ 23 
l.UJ_o 21 

27] 

n 79+0.20 
u ‘ /z -0.18 

28] 

0.89 

± 

0.14 

h° — >■ TT 

1 4Q+0- 43 
1 ' 4,3 -0.37 

29] 

O 

00 

1 + 
oo 

to to 

-4 -4 

0 

0.99 

± 

0.22 

h° -)• bb 

n Cjo+O.40 
u ’ oz -0.40 

31] 

1 OD +0 ’ 50 

i - uu -0.50 

32] 

0.71 

± 

0.31 


TABLE I: Signal strengths for the Higgs boson at the LHC. The last column is our combination. 
The combined signal strength for h° —> WW* + ZZ* (VV) is p(VV) = 0.96 ± 0.12. 


and gauge bosons normalized to their Standard Model values [33]. These scale factors can 


then be expressed in terms of the parameters of a specific 2HDM. 

Negative results from heavy Higgs searches provide us further insights on the parameter 
space of an extended Higgs sector. One of the strongest results comes from the search for a 
heavy Higgs decaying into the WW and ZZ final states, excluding a heavy Higgs boson with 
SM like couplings all the way up to 1 TeV 34[. We use these results to further constraint 


the parameter space of 2HDM. 

In Fig. m we present the 68% (95%) confidence level (C.L.) regions in dark (light) color 
that are compatible with LHC constraints from the light Higgs boson (h°) alone as well as 
constraints from both the light Higgs boson and the heavy Higgs boson (H°) decaying into 
WW and ZZ j34| for a general 2HDM and and a Type-II 2HDM. This figure shows that 


a large value of j cos(^ — a) | is still a possibility within a general model for p tt < 1. This 
is due to the lack of a strong constraint on the 6-quark coupling of the Higgs boson. To be 
consistent with the SM Higgs cross section from gluon fusion, a small value of cos(/? — a) 
is favored for p tt 1 with A htt ~ = Ktt- Experimental data from Run 2 with higher 

energy and higher luminosity will provide much better guidance for parameters such as ptt 
and cos ((3 


a) 


B. Constraints from B Physics 

The FCNH coupling p ct affects the H + tq couplings (q = d, s, b ) through (p u ^V) tq = 
Pt t Vtq + p* ct Vcq + PutVuq • This effect contributes to FCNC processes in down-type quark 
sector via H + and t loops. For simplicity, let us assume p ut is negligible. 

Recasting the 2HDM-II expression |35|, we estimate the modifications to the B q -B q (q = 
d, s ) mixing amplitude (Mf 2 ) from the box diagrams with internal H + /W and t by 

Mb iwH(y w ,y H ,x) + Ihh(v h ) /.x 

[m? 2 ] sm “ Iww(y w ) ’ 1 J 
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(a) Constraints from h° (b) Constraints from h° & H° 



cos(/3 -a) cos(/?-a) 


FIG. 1: Favored regions in cos (/3 — a)-ptt plane at 68% (95%) C.L. of LHC Higgs data in a general 
2HDM [green, (light cyan)] and Type II 2HDM [red, (light magenta)], with constraints from (a) 
light Higgs boson (h°), (b) both light Higgs boson (h°) and heavy Higgs boson (H°). 


where y l = m 2 /m 2 (i = W, H + ), x = m 2 H+ /ml v , and 

,w \ 3 


I\vw — 1 + 
IWH ~ 
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1 -y w (1 -y w ) 2 y w \i-y w 
( Pu VcbP* ct \ ( pu v: q pa\ H 
\ Kt V tb K t )\ K t V t * q K t ) ^ 
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_(l - x)(l - y 11 ) 2 ' (1 -x)(l -y w ) 2 (l-y w )(l-y H ) 


Ihh - 


Pu VcbP*ct\ (Pu V c * q Pct\ H 

Kt VtbKt ) \Kt V t * q K t ) 


1 + y H 2y H In y H 


(i -y H ) 2 (i -y H Y 


(5) 


We adopt the following intervals from the Summer 2014 results by UTht 36], 


C Bd e [0.76, 1.43], cj )Bd e [-8.0°, 4.4°], 

C Bs e [0.9, 1.23], (P Bs e [-3.0°, 4.7°], (6) 

at 95% probability, where C Bq e 2l ^ B i = M® 2 /[Mf 2 ]sM- 

The constraints from Bd }S mixing data are shown in Fig. [2](a) on the (ptt, Pet) plane with 
m H + = 500 GeV. Shaded regions are excluded by the 95% probability ranges in Eq. ([6]). The 
constraint from C Bg (pink regions) is slightly tighter than the C Bd exclusion (blue-shaded 
regions). Combining them with constraint from the CP-violating phase (f) Bd (light-green 
regions), we obtain the upper limit \p tt \ < 1.5, regardless of p ct . The parameter p ct is strongly 
constrained since its effect in Eq. (JSJ) is enhanced by the CKM factor \V cq /V tq \ 25 (q = d,s). 
Once p tt is hxed within this range, we obtain a constraint on p ct . For 0.5 < \p tt \ < 1.5, we 


5 




































have || < 0.06. Furthermore, the sizable phase in V c d/V t d makes p ct sensitive to the CP- 
violating phase (j)B d , even if p ct is real. For m#+ = 300 (700) GeV, the constraints become: 
\ptt\ < 1.2 (1.8) regardless of p ct , and |p c t| < 0.05 (0.09) for 0.5 < \p tt \ < 1.2 (1.8). 



Ptt Pn 

FIG. 2: Allowed regions in (a) ptt~Pct plane from Bd iS -mixing, with shaded regions excluded by 
Eq. ©, i.e. blue (pink) regions by Cs d , s y light-green regions by <j>B d ; (b) ptt~Pbb plane from b —» sy 
with p ct = 0 and m H + = 500 GeV, with dark-green shaded regions excluded by Rex p S7 in Eq. (jHJ) 
at 2a, while pink regions by Cb s ■ Blue-solid lines in (b) mark \pbb\ = « b with rrib{p = mt). 


We now turn to the b —y sy constraint. The H + -t loop affects this process via the Wilson 
coefficients CV )8 at leading-order (LO), which are, at the matching scale p 0 , given by 


5C„ ~ I (^ + (^ + p% H ) 

3 \ Kt VtbKt / \ K t Vts K t J 


Ptt VcsPcA Pbb 77,(2)/ H 
W,8 \V 


Kt + V t * s K t 


Kb 


(7) 


Here, the operator basis and the definition of F^\y) follow Ref. 
procedure in Ref. 38] and calculate first the ratio 


37 . We follow the 


77)fc— B{B —> W,7) exp 

exp B(B^X sl ) s M ’ U 

with world average of measurements B{B — > -V s y) exp = (3.43 ± 0.21 ± 0.07) x 10~ 4 [39| and 
the next-to-next-to LO (NNLO) prediction in SM, B[B —> -V s y)sM = (3.15 ± 0.23) x 10~ 4 
fioj ]. We then require the similar ratio based on our LO calculation, i.e. = R(R —> 

X s1 ) 2W m/B(B —> X s y)sM, to be within the range allowed by R b ex p S7 . Choosing the matching 
scale and the low-energy scale as p 0 ~ m t and pb ~ wrib/ 2, we reproduce the NNLO results 
of Ref. [4l|, rn,H+ > 380 GeV, with 95% C.L. in the Type-II 2HDM 1 . 

Fig. [2](b) shows the allowed region in the (pu, Pbb) plane from b —* sy and B mixing 
for m H + = 500 GeV and p ct = 0. The dark-green shaded regions are excluded by the 2a 


Recent NNLO calculation 


42] provides a stronger limit [m H + > 480 GeV) at 95% C.L. 
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experimental error of in Eq. (l 8 lh with the theoretical uncertainty linearly added. The 

constraint on p u by Cb s is also shown. For p tt ~ K t ~ 1, pbb is constrained to be within 
—0.02 < p^ < 0.01. Note that this touches the region of \pbb\ ~ Kb ~ 0.02. We set p ct = 0 
as it is already strongly constrained by B ds mixing. For m H + = 300 (700) GeV, the b —> sy 
constraint on p b b becomes: —0.009 (—0.03) < pbb % 0.008 (0.02) for p tt ~ 1. Typically, 
B(B —> Xg'y) constrains pbb more strongly than p tt , as the effect of p^ is enhanced by the 
chiral factor K t /K b = m t /m b in Eq. (0. 

The contribution from charm loop has a mild dependence on p tc through the H + cou¬ 
plings, (p ut V) cq = p* tc V tq + p* cc V cq + p* uc V uq (q = d, s, b). In general, p tc may be very different 
from p ct . Since the charm quark in the loop is light and there is no CKM enhancement, the 
constraint on p tc is expected to be much weaker. The constraint on p tc has been analyzed in 
Ref. |2gj. With —632 = ptc sin/3, it is found from B s mixing that |eg 2 1 < 1.7 for m#+ = 500 
GeV and ta n/3 = 50. This constraint implies that \p tc \ < 1.7 for niH+ = 500 GeV. We note 
that a large p tc can enhance B —>■ D^tv rates via the H + coupling (p u ^V) r h and can explain 
the 3.4cr discrepancy between the SM prediction and BaBar data |43l-l45l] . For real valued p tc 
and p TT , the solution space shown in Ref. jH] reads p tc ~ 0.7 x (—0.5/p rr )(mu+/500 GeV) 2 . 


Additional flavor constraints can be obtained from K — K mixing ( p ct < 0.14) 38] and D — D 


mixing (\ptcPt u \ 0.02) [38| for mu — 'm H + = 500 GeV. The value of ( p tu ) is expected to be 
very small, thus B — B mixing provides a better limit for p tc - 

Combining experimental limits from LHC Higgs data and B physics, and assuming per- 
turbativity, we consider p tt < 2, p tc < 1.5, and p ct < 0.1. 


III. SIGNAL AND BACKGROUND 

We now discuss the prospects of discovering FCNH interactions at the LHC through 
H° and A 0 decays. The number of free parameters in a general 2HDM is too large for a 
comprehensive collider study of the FCNH signal, so we make some assumptions that are 
motivated by experiment. The latest experimental results point to a Higgs sector with the 
light CP even state behaving like the SM Higgs, indicating that cos (/3 — a) should be small. 
We consider sample cases with cos(/3 — a) = 0.1 and 0.2, which imply sin(/3 — a) ~ 1. 

In our case study, we choose the heavier states ( H °, A 0 and Af ± ) to be degenerate for 
simplicity, which is also in accordance with the decoupling limit [l9j, and we set Agy = 0 in 
the Higgs potential [ 9 J. We also set tan (3 — 1 and choose m\ 2 such that the scalar potential 
satisfies stability, tree-level unitarity, and perturbativity up to large masses. 

To fix the Yukawa couplings, we assume that p tt = Kt while p bb = K b . This is in good 
agreement with both B physics constraints as well as LHC Run 1 constraints. For the off- 
diagonal parts that generate the flavor changing signal, p ct is constrained to small values 
by B physics but we assume that p tc can have larger values. I 11 the massless limit for 
charm, the signal cross section is, to a very good approximation, only a function of a single 
effective coupling p tc = -^\fp^pr pl t > but also very weakly depends on -^\/Pt c ~ Pct- The 
contribution to the cross section from terms with p tc is at least 98% without cuts and more 
than 93% with all cuts for pp —> H —> bc£u + X with run = 1 TeV, and it is even more 
dominating for a lower Higgs mass. 

With these experimentally motivated choices, gluon fusion is the dominant production 
mode for H° and A 0 states, and tt becomes the dominant final state at high mass (2 rrit < 
1710 < 2 TeV). We display in Fig. [3] the branching fractions for (a) the heavier scalar H°, 
and (b) the pseudoscalar A 0 , as functions of Higgs mass with cos(/3 — a) = 0.1 and p tc = 
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0.24. The computer code 2HDMC 46] is employed to scan over |mi 2 | < 2 TeV and 0.1 < 


tan/3 < 50 with sets of parameters that satisfy potential stability, tree-level unitarity, and 
perturbativity. This gives rise to the “bands” in Fig. 3(a). We also display the branching 
fraction B(H° —» tc ) for the aforementioned choice of tan/3 and m\ 2 in our LHC case study 
with a dashed curve in Fig. 3(a). We note that with large branching fraction in most of the 
parameter space, H° —> h°h° might offer great promise to discover Higgs pairs at the LHC. 



m H (GeV) 


nh (GeV) 


FIG. 3: Branching fraction of (a) heavier Higgs scalar H° and (b) Higgs pseudoscalar versus 
rritj,, with cos(/3 — a) = 0.1, ptc = 0.24, and pa = Ki for diagonal couplings. We show the allowed 
regions when tan/3 and m\ 2 are varied. Branching fraction B(H° —>• tc) for the LHC case study is 
shown as a dashed curve. 


A. Higgs Signal 

Our signal is the production of a heavy Higgs boson from gluon fusion, with subsequent 
flavor changing decays into a charm quark and a top quark, and the top decays semileptoni- 
cally. More explicitly, we consider gg ^ 0° —>■ tc + tc (</>° = H° or A 0 ), followed by tc —> blue 
with i = e or g. LInless explicitly specified, q generally denotes a quark ( q ) or an anti-quark 
( q ) and £ represents a lepton (£~) or anti-lepton (£ + ). We calculate the matrix elements 
analytically, and compute the signal cross section with the parton distribution functions of 
MSTW2008 [47j. The factorization and renormalization scales are chosen to be Pf,r = 

In addition, to estimate the JNLO cross section for pp —> 0° —)■ tc + tc —)■ blue + X, we use 
the computer code HIGLU (48| to calculate a{jpp —> 0° + X) (</>° = H°, H°), including both 
top and bottom quark loops to find a A"-factor. 


B. Standard Model Background 

The dominant physics background to the final state of bjlu comes from Wjj + Wbb, as 
well as s- and f-channel single top (tb + tj). Another important background is tt production 
where either one of the two leptons is missed for both top quarks decaying semileptonically, 
or two of the four jets are missed when only one of top quarks decays semileptonically. 


















We employ the programs MadGraph [49|, |50] and HELAS |51] to evaluate the exact matrix 


elements for the background processes. The factorization and the renormalization scales are 
chosen to be = ™>w for Wjj and Wbb, pr } f = for s- and 6-channel single top, and 
Hr,f = for tt. We use MCFM j52j to calculate the NLO A^-factors for our background 
processes. 


C. Mass Reconstruction 


Let us present our strategy for full reconstruction of each event with the help of inter¬ 
mediate on-shell particles. For each event, we require one b jet and one non-6 jet, identified 
through 6-tagging. In addition, we require a single isolated lepton and missing transverse 
energy from the neutrino in the semileptonic decay of the top quark in our FCNH signal. 
For lepton momentum p and neutrino momentum k, the invariant mass constraint for an 
on-shell W, (k + p) 2 = m^, can be solved for the longitudinal component of the neutrino 
momentum (k z ), which is the only unknown in the event. We obtain two solutions 


, ± _ p 2 (2k T • P t + m 2 w - rrij) ± E e A A 2 
2(mj +Pt) 


(2k r • Pt + rriw - m 2 ) 2 - Ak^(m 2 + p 2 T ) . (9) 


If A 2 < 0 hence kf complex, the event is vetoed. For A 2 > 0 with kf real, we choose the 
solution that minimizes the reconstructed top mass | M h(u — m t \ or |(pb + p + k) 2 — m 2 1. 

Systematics can be the limiting factor for new physics searches at high luminosities. 
Precise determination of the background needs to include systematics in experiments. Since 
our signal is a sharp peak over a smoothly falling background, the precise knowledge of 
background cross section at percent level is not required for a 5a discovery. An uncertainty 
of 30% in the background estimation might shift the limit on gRtc by 10% without affecting 
our results. 


D. Realistic Acceptance Cuts 


To study the discovery potential, we employ three sets of realistic cuts and tagging 
efficiencies. For low luminosity (LL): (a) LHC Run 1 (yfs = 8 TeV) with L =25 fb' 1 (53|; 
and (b) full CM energy (yfs = 13 or 14 TeV) with L = 30 fb -1 . For hi gh luminosity (HL): 
(c) full CM energy (^/s = 14 TeV) with L = 300 fb -1 or 3000 fb -1 (U [55| . 

We require that in every event there should be (a) exactly 2 jets that have pt > 20 GeV 
(30 GeV for HL) and \r}\ < 2.5, and one of them must be tagged as a 6-jet; (b) exactly one 
isolated lepton that has px > 20 GeV and \rj\ < 2.5; (c) at least 20 GeV (40 GeV for HL) 
of missing transverse energy (^ T ). After reconstructing the longitudinal component of the 
neutrino momentum, we further require that (d) the reconstructed invariant mass of the 
top satisfies |muu — m t \ < 0.2 m t ] (e) the reconstructed invariant mass of the Higgs boson 
satisfies \muuc — < 0.2 

We consider a further powerful acceptance cut on non-6-tagged jet momentum. In the 
Higgs boson decay frame, the charm quark momentum from H°, A 0 —> tc is approximately 
given by 


Pc 


m ,^ 

2 


ml 


( 10 ) 
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Since the Higgs boson from gluon fusion has little transverse momentum, the Pt(c) distri¬ 
bution has both a kinematic cut-off and a peak at the above p c value. We require that the 
transverse momentum of the non-5-tagged jet satisfies 0.85 p c <Pr(c ) < 1.10 p c . 

To simulate detector effects based on ATLAS (HI] and CMS 55] specifications, we apply 
Gaussian smearing of momenta: 


A E 


0.60 


a/ A(GeV) 


© 0.03 (jets), and 


A E 
~E~ 


0.25 


V 7 A(GeV) 


0.01 (leptons), (11) 


with individual terms added in quadrature (©). In LHC Run 1 at a/s = 8 TeV, the 5-tagging 
efficiency (e&) is taken to be 50%, the probability that a c-jet is mistagged as a 5-jet (e c ) is 
14% and the probability that any other jet is mistagged as a 5-jet (e/ is taken to be 1%. 
At the full CM energy (y^ = 13 or 14 TeV) with L = 30 fb -1 , we follow the tagging and 
mistag efficiencies in the ATLAS Technical Design Report jHij]: e& = 60%, e c = 14% and 
€j = 1%. For the full CM energy (a/s = 13 or 14 TeV) with HL of 300 fb -1 or 3000 fb -1 , 
the tagging and mistag efficiencies are taken to be e& = 50%, e c = 14% and €j = 1%. 




m H (GeV) 


m H (GeV) 


FIG. 4: The cross section of the heavier Higgs scalar H° (solid) a(pp —>• H° —>• tc + tc —» 
bjl + Ip T + X) at the LHC versus mu for (a) y/s = 8 TeV and (b) y/s = 14 TeV, with p tc = 0.24, 1 
and cos(/3 — a) = 0.1. Also shown are the background cross sections (dashed) from single top (tb 
and tj), IF+jets (Wjj and Wbb) and tt with A'-factors, acceptance cuts, and tagging efficiencies. 


IV. DISCOVERY POTENTIAL 

We present the signal and background cross sections at the LHC for y/s = 8 TeV and 14 
TeV in Fig. |4j All tagging efficiencies and A'-factors discussed above are included. We ob¬ 
serve that the largest contributions to the SM background come from single-top and W ©jets 
processes, which is to be expected, since they can both produce very similar kinematics to 
our signal process. In contrast, the ti background is substantially lower because of the 
requirement on the number of jets and leptons passing our cuts. 
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(a) H° alone (b) Degenerate H° and 4° 




mr(GeV) 


/ih( GeV) 


FIG. 5: Discovery reach at 5a in the m^-ptc plane for the pp —» 0° —>• tc + fc — > bji + Ifi+ X signal 
at the LHC with yfs = 13 (14) TeV for dashed (solid) contours, (a) is for the heavier Higgs scalar 
(H°) and (b) for the combined and A 0 assuming degeneracy, both for cos ((3 —a) = 0.1. (c) and 
(d) are analogous, but for cos(/3 — a) = 0.2. The discovery region is the parameter space above the 
contours. Also shown is the future ATLAS sensitivity at 95 % confidence level for t —>• chP —» C77. 


To estimate the discovery potential, we obtain the lower limit on as by requiring that the 
99.4%-confidence-level (CL) upper limit on the background is smaller than the 99.4%-CL 
lower limit on the signal plus background [56j with statistical fluctuations. This leads to the 
condition, 



N + 2 a/ La b 1 


( 12 ) 


where as(B) is the signal (background) cross section and L the integrated luminosity. Choos¬ 
ing the parameter N = 2.5 corresponds to 5 a significance. For a large number of events 
(Las 3> 1), this requirement is equivalent to the statistical significance 


Nss 


Ns _ La s 
VNb VLag 
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where N$(b) is the number of signal (background) events. 

We show in Fig. [5]the discovery reach in the m^-ptc plane for the FCNH signal pp —?• 0° —?• 
tc T tc —y bj£ + Ip at the LHC with y/s = 13 (14) TeV, for dashed (solid) contours and for 
cos(/3 — a) = 0.1 and 0.2. Fig. [5](a,c) are for the heavier scalar H° alone, whereas Fig. E](b,d) 
are for the degenerate case, for which the scalar H° and pseudoscalar A 0 signals are added 
together. The FCNH decay of the heavy Higgs will be observable for cos(/3 — a) = 0.1 and 
p tc = 0.1 up to m H > 800 GeV with 3000 fb -1 data. A larger value of p tc will enhance the 
cross section, hence statistical significance, of this FCNH signal. 


V. CONCLUSION 

In a general two Higgs doublet model, there could be flavor changing neutral Higgs 
interactions with fermions. Strong limits exist for these FCNH interactions, except those 
involving the third generation quarks. It is of great interest to study the relation between 
the most massive elementary particle (the top quark) and the Higgs bosons. The LHC 
has discovered a Higgs boson lighter than the top, which makes the rare decay t —y ch° 
kinematically possible. In a general 2HDM, the decay width of t —> ch° is proportional 
to cos (/3 — a), while that of H° —y tc is proportional to sin(/3 — a). Therefore, they are 
complementary to each other in the search for new physics beyond the Standard Model. 

We investigated the prospects for discovering H°, A 0 —y tc at the LHC, where the heavy 
scalar H° and pseudoscalar A 0 are produced via gluon fusion, which are facilitated by the 
extra tt couplings. The primary physics background comes from Wjj , tj, Wbb, tb, and 
tt. Both signal and background processes are studied with realistic acceptance cuts as well 
as tagging and mistag efficiencies. Promising results have been found for the LHC with a 
center of mass energy of 13 TeV and 14 TeV. The FCNH decay of the heavy Higgs will 
be observable for cos(/3 — a) = 0.1 and p tc = 0.1 up to M H = 800 GeV with 3000 ffiu 1 of 
integrated luminosity. This result is robust against a small cos ((3 — a), independent of the 
t —> ch° search, which becomes diminished. If c-tagging efficiency can be improved [57} , the 
discovery potential of this FCNH signal will be greatly enhanced. 

If p tc > 0.5, B(H° —* tc) can become comparable to B(H° —> tt) or surpass it. Recently, 
it was suggested that next-to-leading order QCD and electroweak corrections might swamp 
the signal of Higgs decays into top quark pairs [5sj . A recent analysis shows that H° —y tt 
with SM couplings can be very difficult to observe at the LHC [59']. In that case the FCNH 
decay of H°, A 0 —y tc + tc might offer a promising opportunity to observe the heavier Higgs 
bosons. 

We have not emphasized the r lepton sector. Recently, the CMS Collaboration reported 
unexpected rp, events [60] that might be explained by neutral Higgs boson decay [U]. If 
this can be confirmed by the ATLAS collaboration in the near future, or at LHC Run 2, it 
will be exciting new physics for FCNH interactions, and H°,A° —> T±pA, unsuppressed by 
decoupling (i.e. small cos(/5 — a)), could help discover the exotic scalars. 
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